Fix bugs from the fourth bug hunt: invisible characters, dates, numbers, CSV, CLI - #93

Fourth bug hunt. It had three parts:

• a fuzzer for invisible characters and spelling variants (NBSP, soft hyphen, zero-width characters, BOM, decomposed letters, apostrophe variants) and for the case forms of adjective surnames;
• an audit of dates and identifiers;
• an audit of the CLI and of unmasking with unusual files.

All entries are in CHANGELOG.md under [Unreleased].

Data left unmasked

• Invisible characters inside a surname. Soft hyphens, zero-width spaces, a BOM in the middle of the text, and ‘ or ` typed instead of the apostrophe (ДЕМ‘ЯНЧУК) are common in text copied from Word, PDF or the web.
  • Before, the surname-plus-initials phase masked only the tail of such a surname (ЗІНЧЕНКО → ЗІНЧЕНАНЕЦЬ), and the main parser then masked the whole word again, so unmasking could not restore it.
  • These characters are now part of the word in every phase. The mask is made from the clean word, and the mapping keeps the spelling of the document.
• CSV cells. Cells separated by a comma with no space (Петренко,Іван,Петрович) and quoted cells ("Петренко";"Іван", "Петренко","Іван") left the name open; they now separate words.
• Dates of laws. The legal-act exception matched законодавства, законних and any від after a comma (на підставі закону, рапорт від 01.02.2025), so those dates stayed open.
  • It now applies only right after the name of an act. The act's number, України, a quoted title and a line break may come in between.
  • The date of a law with a long quoted title, or with від on the next line, is now kept as well (before, it was shifted).
• Numbers:
  • БР566 and БР: 566 are now masked.
  • Every number in накази № 12, 13 та 14 is masked, not only the first.
  • наказ № 15.01.2025 is masked as a date.
  • в/ч А 1234 with a non-breaking space and в/ч А-1234 are now masked.
  • With --only passport, old passports (АА123456) are masked.
  • Text dates in single quotes are now recognised.

Unmasking did not restore the original

• A one-digit order number (№ 5) in a document with open numbered items 1–9 got a mask equal to an item number, and unmasking swapped them. It now gets a two-digit mask when every one-digit mask is taken.
• A text date with no space before the month («28»вересня 2025) was masked but not restored.
• In a double surname with an invisible character after the hyphen, the second part came back in lower case.

Masks

• У 2025 році, В 2024 р., А 2000 грн were masked as military unit numbers (У 9309 році). A spaced letter-and-digits form is now a unit number only after в/ч or частина.
• Soft adjective surnames (Задорожній / Задорожнього / Задорожнім) and feminine datives (Головатій) now share the stem of the other case forms.
• A text date now always changes the day; before, the 1st and the 28th could keep it.

Command line

• --log-file naming the input, the output or the mapping is now an error (exit code 2). Before, log lines were written into the original document or the mapping.
• data-unmask used to accept any JSON object as a mapping and report success. It now rejects:
  • files without mappings;
  • masks made only of spaces or punctuation;
  • a malformed re_mask_passes.
• A mapping saved with a BOM is now read.
• The size limit for a mapping is now 30× the configured input limit; before, it was a fixed 100 MB.
• In automatic mode, -o and --map are no longer ignored.
• These used to give a traceback and are now errors:
  • non-text codecs (base64, hex, rot13);
  • too deeply nested JSON;
  • -o naming a directory or a missing directory.
• scrypt parameters in a forged .enc file are now limited by total work.

Performance

• Text dates are no longer searched in quadratic time on long runs of spaces (80 000 spaces took 4 minutes).
• Files with only CR line breaks, or with U+2028/U+2029, were processed as one long line. Their lines are now processed one by one.
• Compiled word patterns are cached, so the wider character classes cost no extra time.

Tests

• New tests/test_fuzz_findings4.py with 79 tests. It does not import on main (it needs the new clean_word).
• Full suite: 1886 passed, 1 skipped. mypy and flake8 are clean.
• Fuzzing after the fixes, about 3500 documents across the earlier and new fuzzers:
  • round trip: 0 failures;
  • --re-mask chains: 0 failures in 320;
  • documents of 500–8000 lines with 300 people all restore. An 8000-line document masks in 20.7 s, against 21.4 s before.
• The only leaks the fuzzers still report are two-word names split across lines, a known limitation.

Known limitations, not fixed

• Surnames with vowel alternation (Кравець/Кравця, Лебідь/Лебедя, Соловей/Солов'я) still get a different stem per case.
• The start of a date range without a year (з 01.02 по 03.02.2025) stays open.
• JSON with the parts of a name in separate fields is not masked.
• Unmasking time grows with the number of masks.

Generated by Claude Code